{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Isolation Forest (IF) outlier detector deployment"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Wrap a scikit-learn Isolation Forest python model for use as a prediction microservice in seldon-core and deploy on seldon-core running on minikube or a Kubernetes cluster using GCP."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Dependencies"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- [helm](https://github.com/helm/helm)\n",
    "- [minikube](https://github.com/kubernetes/minikube) \n",
    "- [s2i](https://github.com/openshift/source-to-image) >= 1.1.13 \n",
    "\n",
    "python packages:\n",
    "- scikit-learn: pip install scikit-learn --> 0.20.1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Task"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The outlier detector needs to detect computer network intrusions using TCP dump data for a local-area network (LAN) simulating a typical U.S. Air Force LAN. A connection is a sequence of TCP packets starting and ending at some well defined times, between which data flows to and from a source IP address to a target IP address under some well defined protocol. Each connection is labeled as either normal, or as an attack. \n",
    "\n",
    "There are 4 types of attacks in the dataset:\n",
    "- DOS: denial-of-service, e.g. syn flood;\n",
    "- R2L: unauthorized access from a remote machine, e.g. guessing password;\n",
    "- U2R:  unauthorized access to local superuser (root) privileges;\n",
    "- probing: surveillance and other probing, e.g., port scanning.\n",
    "    \n",
    "The dataset contains about 5 million connection records.\n",
    "\n",
    "There are 3 types of features:\n",
    "- basic features of individual connections, e.g. duration of connection\n",
    "- content features within a connection, e.g. number of failed log in attempts\n",
    "- traffic features within a 2 second window, e.g. number of connections to the same host as the current connection\n",
    "\n",
    "The outlier detector is only using 40 out of 41 features."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Train locally\n",
    "\n",
    "Train on small dataset where you roughly know the fraction of outliers, defined by the \"contamination\" parameter."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "# define columns to keep\n",
    "cols=['duration','protocol_type','flag','src_bytes','dst_bytes','land',\n",
    "      'wrong_fragment','urgent','hot','num_failed_logins','logged_in',\n",
    "      'num_compromised','root_shell','su_attempted','num_root','num_file_creations',\n",
    "      'num_shells','num_access_files','num_outbound_cmds','is_host_login',\n",
    "      'is_guest_login','count','srv_count','serror_rate','srv_serror_rate',\n",
    "      'rerror_rate','srv_rerror_rate','same_srv_rate','diff_srv_rate',\n",
    "      'srv_diff_host_rate','dst_host_count','dst_host_srv_count','dst_host_same_srv_rate',\n",
    "      'dst_host_diff_srv_rate','dst_host_same_src_port_rate','dst_host_srv_diff_host_rate',\n",
    "      'dst_host_serror_rate','dst_host_srv_serror_rate','dst_host_rerror_rate',\n",
    "      'dst_host_srv_rerror_rate','target']\n",
    "cols_str = str(cols)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Load dataset\n",
      "\n",
      "Generate training batch\n",
      "\n",
      "Train outlier detector\n",
      "\n",
      "Training done!\n"
     ]
    }
   ],
   "source": [
    "!python train.py \\\n",
    "--dataset 'kddcup99' \\\n",
    "--samples 50000 \\\n",
    "--keep_cols \"$cols_str\" \\\n",
    "--contamination .1 \\\n",
    "--n_estimators 100 \\\n",
    "--max_samples .8 \\\n",
    "--max_features 1. \\\n",
    "--save_path './models/'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Test using Kubernetes cluster on GCP or Minikube"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Pick Kubernetes cluster on GCP or Minikube."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "minikube = False"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Fetching cluster endpoint and auth data.\n",
      "kubeconfig entry generated for standard-cluster-1.\n"
     ]
    }
   ],
   "source": [
    "if minikube:\n",
    "    !minikube start --memory 4096 --feature-gates=CustomResourceValidation=true \\\n",
    "    --extra-config=apiserver.Authorization.Mode=RBAC\n",
    "else:\n",
    "    !gcloud container clusters get-credentials standard-cluster-1 --zone europe-west1-b --project seldon-demos"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Create a cluster-wide cluster-admin role assigned to a service account named “default” in the namespace “kube-system”."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "clusterrolebinding.rbac.authorization.k8s.io/kube-system-cluster-admin created\r\n"
     ]
    }
   ],
   "source": [
    "!kubectl create clusterrolebinding kube-system-cluster-admin --clusterrole=cluster-admin \\\n",
    "--serviceaccount=kube-system:default"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "namespace/seldon created\r\n"
     ]
    }
   ],
   "source": [
    "!kubectl create namespace seldon"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Add current context details to the configuration file in the seldon namespace."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Context \"gke_seldon-demos_europe-west1-b_standard-cluster-1\" modified.\r\n"
     ]
    }
   ],
   "source": [
    "!kubectl config set-context $(kubectl config current-context) --namespace=seldon"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Create tiller service account and give it a cluster-wide cluster-admin role."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "serviceaccount/tiller created\n",
      "clusterrolebinding.rbac.authorization.k8s.io/tiller created\n",
      "$HELM_HOME has been configured at /home/arnaud/.helm.\n",
      "\n",
      "Tiller (the Helm server-side component) has been installed into your Kubernetes Cluster.\n",
      "Happy Helming!\n"
     ]
    }
   ],
   "source": [
    "!kubectl -n kube-system create sa tiller\n",
    "!kubectl create clusterrolebinding tiller --clusterrole cluster-admin --serviceaccount=kube-system:tiller\n",
    "!helm init --service-account tiller"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Check deployment rollout status and deploy seldon/spartakus helm charts."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Waiting for deployment \"tiller-deploy\" rollout to finish: 0 of 1 updated replicas are available...\n",
      "deployment \"tiller-deploy\" successfully rolled out\n"
     ]
    }
   ],
   "source": [
    "!kubectl rollout status deploy/tiller-deploy -n kube-system"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "NAME:   seldon-core-crd\n",
      "LAST DEPLOYED: Tue Dec  4 14:53:17 2018\n",
      "NAMESPACE: seldon\n",
      "STATUS: DEPLOYED\n",
      "\n",
      "RESOURCES:\n",
      "==> v1beta1/Deployment\n",
      "NAME                        DESIRED  CURRENT  UP-TO-DATE  AVAILABLE  AGE\n",
      "seldon-spartakus-volunteer  1        0        0           0          1s\n",
      "\n",
      "==> v1beta1/CustomResourceDefinition\n",
      "NAME                                         KIND\n",
      "seldondeployments.machinelearning.seldon.io  CustomResourceDefinition.v1beta1.apiextensions.k8s.io\n",
      "\n",
      "==> v1/ServiceAccount\n",
      "NAME                        SECRETS  AGE\n",
      "seldon-spartakus-volunteer  1        0s\n",
      "\n",
      "==> v1beta1/ClusterRole\n",
      "NAME                        AGE\n",
      "seldon-spartakus-volunteer  0s\n",
      "\n",
      "==> v1beta1/ClusterRoleBinding\n",
      "NAME                        AGE\n",
      "seldon-spartakus-volunteer  0s\n",
      "\n",
      "==> v1/ConfigMap\n",
      "NAME                     DATA  AGE\n",
      "seldon-spartakus-config  3     1s\n",
      "\n",
      "\n",
      "NOTES:\n",
      "NOTES: TODO\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "!helm install ../../../helm-charts/seldon-core-crd --name seldon-core-crd \\\n",
    "    --set usage_metrics.enabled=true"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "NAME:   seldon-core\n",
      "LAST DEPLOYED: Tue Dec  4 14:53:19 2018\n",
      "NAMESPACE: seldon\n",
      "STATUS: DEPLOYED\n",
      "\n",
      "RESOURCES:\n",
      "==> v1/ClusterRoleBinding\n",
      "NAME           KIND                                             SUBJECTS\n",
      "seldon-seldon  ClusterRoleBinding.v1.rbac.authorization.k8s.io  1 item(s)\n",
      "\n",
      "==> v1beta1/Role\n",
      "NAME          AGE\n",
      "ambassador    1s\n",
      "seldon-local  1s\n",
      "\n",
      "==> v1beta1/RoleBinding\n",
      "NAME        AGE\n",
      "ambassador  1s\n",
      "\n",
      "==> v1/RoleBinding\n",
      "NAME    KIND                                      SUBJECTS\n",
      "seldon  RoleBinding.v1.rbac.authorization.k8s.io  1 item(s)\n",
      "\n",
      "==> v1/Service\n",
      "NAME                          CLUSTER-IP    EXTERNAL-IP  PORT(S)                        AGE\n",
      "seldon-core-ambassador-admin  10.7.243.27   <nodes>      8877:30877/TCP                 1s\n",
      "seldon-core-ambassador        10.7.244.21   <nodes>      80:30645/TCP,443:32092/TCP     1s\n",
      "seldon-core-seldon-apiserver  10.7.244.66   <nodes>      8080:32315/TCP,5000:32435/TCP  1s\n",
      "seldon-core-redis             10.7.242.105  <none>       6379/TCP                       1s\n",
      "\n",
      "==> v1beta1/Deployment\n",
      "NAME                                DESIRED  CURRENT  UP-TO-DATE  AVAILABLE  AGE\n",
      "seldon-core-ambassador              1        1        1           0          1s\n",
      "seldon-core-seldon-apiserver        1        1        1           0          1s\n",
      "seldon-core-seldon-cluster-manager  1        1        1           0          1s\n",
      "seldon-core-redis                   1        1        1           0          1s\n",
      "\n",
      "==> v1/ServiceAccount\n",
      "NAME    SECRETS  AGE\n",
      "seldon  1        1s\n",
      "\n",
      "==> v1beta1/ClusterRole\n",
      "NAME               AGE\n",
      "seldon-crd-seldon  1s\n",
      "\n",
      "\n",
      "NOTES:\n",
      "Thank you for installing Seldon Core.\n",
      "\n",
      "Documentation can be found at https://github.com/SeldonIO/seldon-core\n",
      "\n",
      "\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "!helm install ../../../helm-charts/seldon-core --name seldon-core \\\n",
    "        --namespace seldon \\\n",
    "        --set ambassador.enabled=true"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Check deployment rollout status for seldon core."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Waiting for deployment \"seldon-core-seldon-cluster-manager\" rollout to finish: 0 of 1 updated replicas are available...\n",
      "deployment \"seldon-core-seldon-cluster-manager\" successfully rolled out\n",
      "deployment \"seldon-core-seldon-apiserver\" successfully rolled out\n"
     ]
    }
   ],
   "source": [
    "!kubectl rollout status deploy/seldon-core-seldon-cluster-manager -n seldon\n",
    "!kubectl rollout status deploy/seldon-core-seldon-apiserver -n seldon"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If Minikube used: create docker image for outlier detector inside Minikube using s2i."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "if minikube:\n",
    "    !eval $(minikube docker-env) && s2i build . seldonio/seldon-core-s2i-python3:0.4-SNAPSHOT seldonio/outlier-if:0.1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Install outlier detector helm charts and set \"threshold\" hyperparameter value."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "NAME:   outlier-detector\n",
      "LAST DEPLOYED: Tue Dec  4 14:54:28 2018\n",
      "NAMESPACE: seldon\n",
      "STATUS: DEPLOYED\n",
      "\n",
      "RESOURCES:\n",
      "==> v1alpha2/SeldonDeployment\n",
      "NAME              KIND\n",
      "outlier-detector  SeldonDeployment.v1alpha2.machinelearning.seldon.io\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "!helm install ../../../helm-charts/seldon-od-if \\\n",
    "    --set model.image.name=seldonio/outlier-if:0.1 \\\n",
    "    --set model.threshold=0.04 \\\n",
    "    --name outlier-detector --set oauth.key=oauth-key \\\n",
    "    --set oauth.secret=oauth-secret \\\n",
    "    --namespace=seldon"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Port forward Ambassador\n",
    "\n",
    "Run command in terminal:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```\n",
    "kubectl port-forward $(kubectl get pods -n seldon -l service=ambassador -o jsonpath='{.items[0].metadata.name}') -n seldon 8003:8080\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Import rest requests, load data and test requests"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(4898431, 53)\n"
     ]
    }
   ],
   "source": [
    "from utils import get_payload, rest_request_ambassador, send_feedback_rest, get_kdd_data, generate_batch\n",
    "\n",
    "data = get_kdd_data(keep_cols=cols) # load dataset\n",
    "print(data.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Generate a random batch from the data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(1, 52)\n",
      "(1,)\n"
     ]
    }
   ],
   "source": [
    "import numpy as np\n",
    "\n",
    "samples = 1\n",
    "fraction_outlier = 0.\n",
    "X, labels = generate_batch(data,samples,fraction_outlier)\n",
    "print(X.shape)\n",
    "print(labels.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Test the rest requests with the generated data. It is important that the order of requests is respected. First we make predictions, then we get the \"true\" labels back using the feedback request. If we do not respect the order and eg keep making predictions without getting the feedback for each prediction, there will be a mismatch between the predicted and \"true\" labels. This will result in errors in the produced metrics."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "request = get_payload(X)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "200\n",
      "{\n",
      "  \"meta\": {\n",
      "    \"puid\": \"1qlvfh1d89upve1707bd9co8d5\",\n",
      "    \"tags\": {\n",
      "    },\n",
      "    \"routing\": {\n",
      "    },\n",
      "    \"requestPath\": {\n",
      "      \"outlier-if\": \"seldonio/outlier-if:0.1\"\n",
      "    },\n",
      "    \"metrics\": [{\n",
      "      \"key\": \"is_outlier\",\n",
      "      \"type\": \"GAUGE\",\n",
      "      \"value\": \"NaN\",\n",
      "      \"tags\": {\n",
      "      }\n",
      "    }, {\n",
      "      \"key\": \"anomaly_score\",\n",
      "      \"type\": \"GAUGE\",\n",
      "      \"value\": \"NaN\",\n",
      "      \"tags\": {\n",
      "      }\n",
      "    }, {\n",
      "      \"key\": \"observation\",\n",
      "      \"type\": \"GAUGE\",\n",
      "      \"value\": 0.0,\n",
      "      \"tags\": {\n",
      "      }\n",
      "    }, {\n",
      "      \"key\": \"threshold\",\n",
      "      \"type\": \"GAUGE\",\n",
      "      \"value\": 0.04,\n",
      "      \"tags\": {\n",
      "      }\n",
      "    }, {\n",
      "      \"key\": \"label\",\n",
      "      \"type\": \"GAUGE\",\n",
      "      \"value\": \"NaN\",\n",
      "      \"tags\": {\n",
      "      }\n",
      "    }, {\n",
      "      \"key\": \"accuracy_tot\",\n",
      "      \"type\": \"GAUGE\",\n",
      "      \"value\": \"NaN\",\n",
      "      \"tags\": {\n",
      "      }\n",
      "    }, {\n",
      "      \"key\": \"precision_tot\",\n",
      "      \"type\": \"GAUGE\",\n",
      "      \"value\": \"NaN\",\n",
      "      \"tags\": {\n",
      "      }\n",
      "    }, {\n",
      "      \"key\": \"recall_tot\",\n",
      "      \"type\": \"GAUGE\",\n",
      "      \"value\": \"NaN\",\n",
      "      \"tags\": {\n",
      "      }\n",
      "    }, {\n",
      "      \"key\": \"f1_tot\",\n",
      "      \"type\": \"GAUGE\",\n",
      "      \"value\": \"NaN\",\n",
      "      \"tags\": {\n",
      "      }\n",
      "    }, {\n",
      "      \"key\": \"f2_tot\",\n",
      "      \"type\": \"GAUGE\",\n",
      "      \"value\": \"NaN\",\n",
      "      \"tags\": {\n",
      "      }\n",
      "    }, {\n",
      "      \"key\": \"accuracy_roll\",\n",
      "      \"type\": \"GAUGE\",\n",
      "      \"value\": \"NaN\",\n",
      "      \"tags\": {\n",
      "      }\n",
      "    }, {\n",
      "      \"key\": \"precision_roll\",\n",
      "      \"type\": \"GAUGE\",\n",
      "      \"value\": \"NaN\",\n",
      "      \"tags\": {\n",
      "      }\n",
      "    }, {\n",
      "      \"key\": \"recall_roll\",\n",
      "      \"type\": \"GAUGE\",\n",
      "      \"value\": \"NaN\",\n",
      "      \"tags\": {\n",
      "      }\n",
      "    }, {\n",
      "      \"key\": \"f1_roll\",\n",
      "      \"type\": \"GAUGE\",\n",
      "      \"value\": \"NaN\",\n",
      "      \"tags\": {\n",
      "      }\n",
      "    }, {\n",
      "      \"key\": \"f2_roll\",\n",
      "      \"type\": \"GAUGE\",\n",
      "      \"value\": \"NaN\",\n",
      "      \"tags\": {\n",
      "      }\n",
      "    }, {\n",
      "      \"key\": \"true_negative\",\n",
      "      \"type\": \"GAUGE\",\n",
      "      \"value\": \"NaN\",\n",
      "      \"tags\": {\n",
      "      }\n",
      "    }, {\n",
      "      \"key\": \"false_positive\",\n",
      "      \"type\": \"GAUGE\",\n",
      "      \"value\": \"NaN\",\n",
      "      \"tags\": {\n",
      "      }\n",
      "    }, {\n",
      "      \"key\": \"false_negative\",\n",
      "      \"type\": \"GAUGE\",\n",
      "      \"value\": \"NaN\",\n",
      "      \"tags\": {\n",
      "      }\n",
      "    }, {\n",
      "      \"key\": \"true_positive\",\n",
      "      \"type\": \"GAUGE\",\n",
      "      \"value\": \"NaN\",\n",
      "      \"tags\": {\n",
      "      }\n",
      "    }, {\n",
      "      \"key\": \"nb_outliers_roll\",\n",
      "      \"type\": \"GAUGE\",\n",
      "      \"value\": \"NaN\",\n",
      "      \"tags\": {\n",
      "      }\n",
      "    }, {\n",
      "      \"key\": \"nb_labels_roll\",\n",
      "      \"type\": \"GAUGE\",\n",
      "      \"value\": \"NaN\",\n",
      "      \"tags\": {\n",
      "      }\n",
      "    }, {\n",
      "      \"key\": \"nb_outliers_tot\",\n",
      "      \"type\": \"GAUGE\",\n",
      "      \"value\": \"NaN\",\n",
      "      \"tags\": {\n",
      "      }\n",
      "    }, {\n",
      "      \"key\": \"nb_labels_tot\",\n",
      "      \"type\": \"GAUGE\",\n",
      "      \"value\": \"NaN\",\n",
      "      \"tags\": {\n",
      "      }\n",
      "    }]\n",
      "  },\n",
      "  \"data\": {\n",
      "    \"names\": [],\n",
      "    \"ndarray\": [0.0]\n",
      "  }\n",
      "}\n"
     ]
    }
   ],
   "source": [
    "response = rest_request_ambassador(\"outlier-detector\",request,endpoint=\"localhost:8003\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "send_feedback_rest(\"outlier-detector\",request,response,0,labels,endpoint=\"localhost:8003\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Analytics"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Install the helm charts for prometheus and the grafana dashboard"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "NAME:   seldon-core-analytics\n",
      "LAST DEPLOYED: Tue Dec  4 14:57:24 2018\n",
      "NAMESPACE: seldon\n",
      "STATUS: DEPLOYED\n",
      "\n",
      "RESOURCES:\n",
      "==> v1/ServiceAccount\n",
      "NAME        SECRETS  AGE\n",
      "prometheus  1        2s\n",
      "\n",
      "==> v1beta1/ClusterRole\n",
      "NAME        AGE\n",
      "prometheus  2s\n",
      "\n",
      "==> v1/Job\n",
      "NAME                            DESIRED  SUCCESSFUL  AGE\n",
      "grafana-prom-import-dashboards  1        0           2s\n",
      "\n",
      "==> v1beta1/Deployment\n",
      "NAME                     DESIRED  CURRENT  UP-TO-DATE  AVAILABLE  AGE\n",
      "alertmanager-deployment  1        1        1           0          2s\n",
      "grafana-prom-deployment  1        1        1           0          2s\n",
      "prometheus-deployment    1        1        1           0          2s\n",
      "\n",
      "==> v1beta1/DaemonSet\n",
      "NAME                      DESIRED  CURRENT  READY  UP-TO-DATE  AVAILABLE  NODE-SELECTOR  AGE\n",
      "prometheus-node-exporter  1        1        0      1           0          <none>         2s\n",
      "\n",
      "==> v1/ConfigMap\n",
      "NAME                       DATA  AGE\n",
      "alertmanager-server-conf   1     2s\n",
      "grafana-import-dashboards  9     2s\n",
      "prometheus-rules           0     2s\n",
      "prometheus-server-conf     1     2s\n",
      "\n",
      "==> v1beta1/ClusterRoleBinding\n",
      "NAME        AGE\n",
      "prometheus  2s\n",
      "\n",
      "==> v1/Service\n",
      "NAME                      CLUSTER-IP    EXTERNAL-IP  PORT(S)       AGE\n",
      "alertmanager              10.7.248.9    <none>       80/TCP        2s\n",
      "grafana-prom              10.7.255.122  <nodes>      80:31668/TCP  2s\n",
      "prometheus-node-exporter  None          <none>       9100/TCP      2s\n",
      "prometheus-seldon         10.7.254.117  <none>       80/TCP        2s\n",
      "\n",
      "==> v1/Secret\n",
      "NAME                 TYPE    DATA  AGE\n",
      "grafana-prom-secret  Opaque  1     2s\n",
      "\n",
      "\n",
      "NOTES:\n",
      "NOTES: TODO\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "!helm install ../../../helm-charts/seldon-core-analytics --name seldon-core-analytics \\\n",
    "    --set grafana_prom_admin_password=password \\\n",
    "    --set persistence.enabled=false \\\n",
    "    --namespace seldon"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Port forward Grafana dashboard"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Run command in terminal:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```\n",
    "kubectl port-forward $(kubectl get pods -n seldon -l app=grafana-prom-server -o jsonpath='{.items[0].metadata.name}') -n seldon 3000:3000\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can then view an analytics dashboard inside the cluster at http://localhost:3000/dashboard/db/prediction-analytics?refresh=5s&orgId=1. Your IP address may be different. get it via minikube ip. Login with:\n",
    "\n",
    "Username : admin\n",
    "\n",
    "password : password (as set when starting seldon-core-analytics above)\n",
    "\n",
    "Import the outlier-detector-if dashboard from ../../../helm-charts/seldon-core-analytics/files/grafana/configs."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Run simulation\n",
    "\n",
    "- Sample random network intrusion data with a certain outlier probability.\n",
    "- Get payload for the observation.\n",
    "- Make a prediction.\n",
    "- Send the \"true\" label with the feedback.\n",
    "\n",
    "It is important that the prediction-feedback order is maintained. Otherwise there will be a mismatch between the predicted and \"true\" labels.\n",
    "\n",
    "View the progress on the grafana \"Outlier Detection\" dashboard."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "import time\n",
    "n_requests = 100\n",
    "samples = 1\n",
    "for i in range(n_requests):\n",
    "    fraction_outlier = .1\n",
    "    X, labels = generate_batch(data,samples,fraction_outlier)\n",
    "    request = get_payload(X)\n",
    "    response = rest_request_ambassador(\"outlier-detector\",request,endpoint=\"localhost:8003\")\n",
    "    send_feedback_rest(\"outlier-detector\",request,response,0,labels,endpoint=\"localhost:8003\")\n",
    "    #time.sleep(1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
